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I .  \  APPROXIMATION  TECHNIQUES  IK  STOCHASTIC  CONTROL  PROBLEMS 

V 

In  this  research  we  continue  our  investigations  of  approxi¬ 
mation  techniques  for  a  wide  class  of  discrete  and  continuous 
time  stochastic  control  problems.  Emphasis  is  placed  on  the 
development  and  theoretical  justification  of  techniques  which 
yield  computationally  tractable  algorithms  that  answer  the 
following : 

(1)  approximations  to  the  optimal  co.-  t  and  the  cost  of 
using  a  particular  control. 

(2)  approximations  to  the  optimal  control. 

(3)  evaluation  of  the  relative  performance  of  two  controls. 

(4)  estimates  for  the  deterioration  :.n  system  performance 
due  to  the  failure  to  observe  syntem  components. 


Small  Noise  Problems  in  Continuous  Tine 


In  many  control  problems  the  noise  entering  the  system  is 
of  low  intensity  reflecting  its  role  as  a  luisance  parameter. 
Recognizing  the  extreme  complexity  of  solving  stochastic  control 
problems,  one  approach  has  been  to  solve  "approximately"  the 
stochastic  control  problem  in  terms  of  quantities  computable  from 
the  optimal  solution  to  the  corresponding  deterministic  control 
problem  which  results  when  the  noise  is  absent,. 

The  general  procedure  in  this  approximation  technique  is  co 
first  establish  expansions  of  the  optimal  cost  and  control  ir. 
powers  of  the  noise  coefficient.  The  expansions  then  suggest 

appropriate  forms  for  nearly  optimal  controls  and  numerical 

AIR  FORCE  OFFICE  OF  SCIENTIFIC  RESEARCH  (AFSCJ 
techniques  for  determining  them  NOTICE  OF  TRANSMITTAL  TO  DDC 

This  technical  report  has  bt.on  reviewed  and  is 
approved  for  publ.c  release  IAW  AFR  190-12  (7b) 
Distribution  is  unlimited. 

A.  D.  BLOSE 

Technical  Information  Offioar 


-4- 


The  theoretical  portion  of  this  reseaich  was  first  treated 
successfully  by  Fleming  [4] .  The  numerical  algorithms  suggested 
by  the  theorems  were  developed  and  employee  on  some  two  dimen¬ 
sional  examples  by  this  author  in  Holland  9] . 

A  partial  solution  to  the  open  loop  control  problem  was 
developed  by  this  author  in  Holland  [9] ,  [  2] .  In  those  papers 

we  were  able  to  theoretically  establish  an  expansion  of  the 
optimal  cost  in  powers  of  the  noise  coefficient..  However,  we 
were  able  only  to  derive  anc  theoretically  justify  an  approxi¬ 
mation  technique  for  the  optimal  control  in  the  restricted  case 
in  which  each  open  loop  control  generates  a  nondegenerate  Gaussian 
process . 

Motivated  by  an  attempt,  to  remove  the  restriction  in  these 
previous  results,  we  recently  considered  in  [17]  more  general 
open  loop  control  problems  in  which  each  open  loop  control  does 
not  necessarily  generate  a  nondegenerate  Gaussian  process.  An 
approximation  technique  was  developed  that  has  the  advantage  that 
one  finds  approximately  optimal  controls  simultaneously  for  all 
sufficiently  small  noise  coefficients.  This  scheme  produces  a 
control  which  performs  better  than  the  use  of  the  optimal  de¬ 
terministic  control  problem  in  the  stochascic  control  problem. 
Moreover  as  we  shew  in  [17] ,  this  scheme  is  superior  to  and  does 
not  agree  with  the  standard  secondary  accessory  control  problem. 

In  this  research  we  intend  to  continue  our  investigations 
into  the  numerical  methods  suggested  by  the  above  schemes.  The 
numerical  method  requires  the  calculation  of  a  generalized  linear 
regulator  problem  which  can  be  solved  numerically.  Attempts  will 
be  made  at  order  reduction  of  th.-r  associat'd  problem. 


-5- 


In  addition,  we  intend  to  look  at  extensions  of  the  above 
techniques  to  sampled-data  problems. 

Finally,  we  are  interested  in  a  class  of  completely 
observable  stationary  control  problems.  Expansions  in  powers 
of  the  noise  coefficient  for  using  a  fixed  control  have  been 
developed  by  this  author  in  [11],  [14]. 

Qualitative  dependence  of  stochastic  control  problems  on  noise 

An  area  of  interest  has  been  the  study  of  the  qualitative 
behavior  of  the  optimal  control  and  cost  of  stochastic  control 
problems.  One  is  not  sure  the  manner  in  which  the  introduction 
of  stochastic  effects  affects  the  behavior  of  the  optimal 
stochastic  control  and  cost.  We  have  done  some  numerical  work 
in  [9]  for  the  completely  observable  problem  on  the  perturbation 
of  switching  surfaces  but  generalizations  are  unable  to  be  made. 

We  recently  discovered  that  a  certain  class  of  stationary 
control  problems  have  the  alternative  interpretation  of  finding 
the  principal  eigenvalue  of  second  order  elliptic,  not  necessarily 
self-adjoint,  partial  differential  equations.  From  the  knowledge 
of  the  principal  eigenvalue  and  eigenfunction  one  can  then  deduce 
the  behavior  of  the  optimal  stochastic  control  and  cost.  This 
then  gives  a  class  of  stochastic  control  problems  for  which  one 
has  an  alternative  way  of  studying  the  qualitative  effects  of 
noise.  We  intend  to  continue  this  investigation  further. 

Fleming  [6]  (and  with  Tsai  [3])  have  been  using  stochastic 
control  methods  which  are  similar  to  ours  to  answer  interesting 
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probabilistic  questions  on  the  physically  significant  exit 
problem.  This  gives  a  second  class  of  problems  to  study  noise 
effects . 

Lastly,  Benes  [1]  has  been  examining  classes  of  problems 
for  which  one  can  compute  explicitly  the  optimal  stochastic 
control . 


4 
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II.  PROBLEMS  IN  OPTIMAL  NONLINEAR  FILTERING  AND  CONTROL 
Stationary  discrete  time  control  problems 

Discrete  time  stochastic  control  problems  have  been  tho¬ 
roughly  considered.  However,  most  prior  work  considers  the  case 
in  which  the  controller  remembers  all  previously  obtained  in¬ 
formation.  This  has  been  called  the  classical  information  pattern 
by  Witsenhausen  [24]  .  We  intend  to  investigate  the  stationary 
control  case  where  the  controller  has  only  partial  observations 
of  the  system  state  and  no  memory.  This  emphasis  is  motivated 
by  systems  in  which  it  istoo  difficult  or  expensive  to  observe  all 
system  components,  and  systems  in  which  it  is  difficult  to 
implement  controls  using  past  information. 

Under  certain  reasonable  assumptions  we  have  succeeded  in 
reducing  the  optimisation  problem  to  a  problem  in  nonlinear  pro¬ 
gramming  [13] .  We  have  also  constructed  examples  to  show  that 
one  can  do  better  by  using  randomized  controls  than  by  only  using 
nonrandomized  controls  of  the  current  observed  data.  In  problems 
of  this  type  with  complete  observations  it  is  known  that  the 
optimization  problem  can  be  treated  as  a  problem  in  linear  pro¬ 
gramming  [22]  and  that  the  controller  cannot  do  better  by  con¬ 
sidering  randomized  controls  [2] . 

Continuous  tine  filtering  problems  4 

In  this  part  of  the  research  we  seek  a  computationally 
convenient  technique  for  solving  the  filtering  and  prediction 
problems  for  a  class  of  nonlinear  stochastic  differential  equations 
subject  to  partial  observations  at  discrete  time  points.  The 
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applications  of  the  technique  described  below  to  trajectory 
estimation  are  apparent. 

Let  there  be  given  q  functions  f f  which  are 

12  q 

known  to  us.  Suppose  for  some  i  6  {1,2,...  q},  unknown  to 
us,  the  state  of  the  process  is  evolving  according  to  the  vector 
stochastic  differential  equations 

dx  =  f^(t,x)dt  +  g(t,x)dv;,  x(o)  -  x. 

Ac  the  discrete  times  t.,...  t  ,  which  are  known  in  advance, 

P 

an  intermediate  observer  (usually  a  machine)  receives  noise 
corrupted  observations  of  the  stochastic  process  x(t).  These 
observations  y(t)  satisfy  the  stochastic  differential  equations 


dy  =  H(t,x,y)dt  +  g(t,x,y)dw,  y(o)  =  0. 


Suppose  that  y(t)  €  Rm  for  some  m.  Assume  there  exists  known 
pairwise  disjoint  sets  Bi,  i  =  1 , . . . ,m,  l  Bi  =  Rm.  Then  at  each 


time  t j ,  j  =  1,2,...  p,  we  receive  from  the  intermediate  observer 
only  the  information  as  to  which  of  the  events  y(t^)  €  B.^  has 
occurred. 


Our  first  problem  is  to  determine  for  each  function  f^,  the 
probability  that  the  function  f ^  is  being  used  given  available 
information . 


Our  second  prediction  problem  is  the  following:  Given  our 

information  at  times  t  =  t  ,t_,...,t  ,  and  the  function  f., 

1  2  p  i 

determine  the  best  prediction  in  mean  square  of  some  function 
h(x(T))  of  the  process,  T  >  t  . 
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Let  us  discuss  problem  2  which  we  have  recently  solved 
theoretically.  For  each  possible  information  set,  we  must  solve 
a  coupled  set  of  second  order  partial  differential  equations. 

Since  the  boundary  conditions  are  of  Cauchy  type,  they  can  be 
solved  numerically.  There  are  m«p  information  sets,  each 
requiring  the  solving  of  the  coupled  set  of  partial  differential 
equations.  Although  these  problems  appear  complex,  they  have 
the  important  practical  advantage  that  they  can  be  completely 
precomputed  and  do  not  need  to  be  solved  in  real-time. 

In  this  research  we  intend  to  consider  the  problems  discussed 
above  looking  at  both  the  theoretical  solution  and  effective 
computational  methods  that  can  be  developed  from  the  theoretical 
solutions . 


4 
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Ili.  SUMMARY  OF  RESEARCH  ACCOMPLISHMENTS 

This  section  summarizes  progress  made  under  AFOSR  Grant  77-3286 
Problems  in  Optimal  Filtering  and  Stochastic  Control.  The  research 
effort  was  concentrated  in  the  following  areas: 

(i)  Approximation  Procedures  for  Open  Loop  Stochastic  Control 
Problems . 

(ii)  Stationary  Stochastic  Control  and  Eigenvalue  Problems. 

(iii)  Long-Term  Effects  of  Noise  on  Dynamical  Systems. 

(iv)  Asymptotic  properties  of  Nonlinear-Diffusion  Equations. 

The  first  three  areas  are  relevant  to  the  control  of  stochastic 
systems.  The  fourth  area  deals  with  stability  properties  and  the  long 
term  behavior  of  systems  described  by  nonlinear  partial  differential 
equations.  No  progress  was  made  in  the  area  of  nonlinear  filtering, 
although  it  should  be  remarked  that  a  transformation  in  area  (ii)  above 
was  developed  and  utilized  independently  by  W.  Fleming,  also  under 
AFOSR  support,  to  study  the  unnormalized  Zakai  equation  arising  in 
nonlinear  filtering. 

Progress  in  each  of  the  above  areas  is  described  in  the  following 


pages. 
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Approximation  Procedures  for  Open  Loop  Stochastic  Control  Problens 

An  Approximation  technique  for  small  noise  open  loop  control 
problems,  Optimal  Control  Applications  and  Methods  2(1981)  89-94. 

Nonlinear  perturbations  of  the  open  loop  stochastic  linear 
regulator  problem,  working  paper. 

A  major  part  of  our  research  effort  has  concentrated  on  the 
development  of  approximation  techniques  for  open  loop  stochastic  control 
problems.  The  basic  theory  for  the  optimal  control  of  Markov  diffusion 
processes  has  been  the  subject  of  extensive  research  over  the  past  decade 
(see  Fleming-Rishel  (2)).  There  remains  the  problem  of  developing 
numerical  procedures  for  computing  the  optimal  control,  or  if  that  is  not 
possible,  approximately  optimal  controls.  If  an  approximately  optimal 
control  is  employed,  one  wants  to  know  the  deterioration  in  performance 
obtained  by  using  this  approximately  optimal  control. 

While  most  deterministic  control  problems  can  be  solved 
numerically,  the  general  problem  of  computing  the  optimal  stochastic 
control  is  impossible  with  present-day  computers.  The  major  exception  is 
the  stochastic  linear  regulator  problem  both  in  the  closed  loop  (completely 
observable)  and  the  open  loop  cases. 

One  is  thus  forced  to  develop  procedures  for  computing 
approximately  optimal  controls  for  most  stochastic  control  problems.  The 
basic  idea  has  been  to  solve  stochastic  control  problems  which  are  in  some 


w 


sense  a  small  perturbation  of  a  control  problem  for  which  the  optimal 
control  can  be  computed  explicitly.  As  was  mentioned  earlier,  two  such 
classes  of  problems  exist: 

a.  deterministic  control  problems. 

b.  stochastic  linear  regulator  problems. 

Both  of  these  classes  of  problems  have  been  exploited  for  the 
completely  observable  case.  First,  Fleming  (1)  developed  the  mathematical 
theory  for  solving  approximately  stochastic  control  problems  in  case  the 
intensity  of  the  noise  entering  the  system  is  small.  He  suggested  a 
numerical  method  for  computing  the  optimal  control  in  terms  of  quantities 
computable  with  knowledge  of  the  optimal  solution  of  the  corresponding 
deterministic  control  problem  which  is  obtained  when  the  small  noise  term 
is  absent.  Holland  developed  the  numerical  procedure  in  (3).  This 
numerical  procedure  has  been  adapted  to  a  model  of  a  problem  of  suboptimal 
control  of  statellite  momenta  in  a  noisy  environment  (see  (4)). 

Second,  Tsai  (5)  developed  the  theory  and  approximation  procedure 
for  the  approximate  solution  of  stochastic  control  problems  which  arise  by 
perturbing  the  stochastic  linear  regulator  problem  through  an  additive  term 
with  a  small  parameter  b  in  the  drift  coefficient  of  the  unperturbed 
dynamical  system.  His  results  show  that  an  approximate  solution  can  be 

4 

computed  in  powers  b,  b^  b^,  ...  of  the  small  parameter  b.  The 
coefficients  of  b,  b^ ,  b^,  ...  are  computable  from  a  knowledge  of  the 
computable  solution  to  the  stochastic  linear  regulator  problem.  Moreover, 
the  results  there  give  an  estimate  of  the  extra  cost  obtained  by  using  the 
approximate  contol. 
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In  this  research  we  have  attempted  to  duplicate  these  previous 
techniques  for  the  stochastic  open  loop  control  problem.  Open  loop 
controls  are  easier  to  implement  than  closed  loop  controls  since  they  do 
not  involve  real  time  processing  of  data.  However,  this  ease  of 
implementation  appears  to  be  partially  mitigated  by  the  extra  complexity  in 
computing  approximate  controls  for  case  b.  above. 

First,  we  treated  the  open  loop  problems  in  case  the  noise  entering 
the  system  is  small.  This  problem  is  the  analogue  of  the  completely 
observable  problem  treated  by  Fleming  (1).  Although  there  is  some  analogy 
with  the  results  of  the  completely  observable  case,  the  methods  required  in 
the  open  loop  case  are  quite  different. 

Our  prior  theoretical  work  in  this  area  had  suggested  that  the 
optimal  control  should  be  expandable  in  powers  of  the  noise  intensity.  An 
approximation  technique  was  thus  sought  that  would  calculate  "best" 
controls  of  a  certain  form.  If  an  expansion  of  the  optimal  control  were 
valid,  the  approximate  control  would  match  the  expansion.  We  were 
successful  with  the  development  of  an  approximation  scheme  for  these 
problems.  The  scheme  requires  the  solution  of  a  generalized  linear 
regulator  problem  (of  determistic  type)  which  is  solved  easily  numerically. 
The  numerical  method  is  given  and  an  example  illustrating  the  efficiency  of 
the  method  is  also  presented  in  the  first  paper  listed  in  the  work  for  this 
section. 


Recently,  we  have  attempted  to  develop  an  approximation  technique 
for  the  approximate  solution  for  the  problem  of  nonlinear  perturbations  of 
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open  loop  stochastic  linear  regulator  problem  (case  b  above).  The  problem 
is  as  follows: 

For  each  open  loop  control  u(t),  the  state  x(t)  evolves  according 
to  the  Ito  stochastic  differential  equation 

dx=  Ax+Bu+bg(x)  dt  +  m(t)dw(t),  x(0)=xq 

where  A,  B  can  depend  upon  t,  and  b  is  a  small  parameter  reflecting  the 
intensity  of  the  noise.  The  cost  of  using  the  control  u(t)  is  given  by 

E  x(t)H(t)x(t)+  u(t)N(t)u(t)  dt. 

The  problem  is  to  choose  the  open  loop  control  u  which  minimizes  the  cost. 

This  problem  is  the  direct  analogue  of  the  completely  observable 
problem  treated  by  Tsai  (5).  His  methods  do  not  apply,  however,  to  the 
open  loop  case.  It  was  shown  for  the  completely  observable  case  that  the 
optimal  feedback  control  Yb  in  the  b  problem  could  be  expanded  as 

Yb  =  Y°+  bZ1+b2Z2+...+bkZk  +o(bk) 

for  any  positive  integer  k  where  the  coefficients  Z^(t,x)  could  be  computed 
with  a  knowledge  of  the  optimal  solution  of  the  linear  regulator  problem. 
Moreover,  his  results  give  an  estimate  of  the  performance  obtained  by  using 
as  an  approximate  control 

Y°+bZ1+...+bkZk 

for  any  positive  integer  k. 
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We  were  interested  in  a  duplication  of  the  analogous  results  for 
the  open  loop  control  problem.  Thus,  if  u*5  is  the  optimal  open  loop 
control,  one  would  want  to  establish  the  above  expansion  with  Y^, 
replaced  by  u*5,  u®  and  the  functions  replaced  by  appropriate  functions. 
We  have  been  unable  to  develop  such  an  approximation  schene. 

However,  we  have  shown  the  following.  Suppose  we  use  any  control 
of  the  form  u^+bV  for  some  function  V.  Then  the  cost  of  using  that  control 
can  be  expanded  as 

C  +  bE  A1  Sx(t,  x°(t))g(x°(t))  dt  ,[  =  S(O,X0) 

where  S(s,x)  is  the  cost  obtained  in  the  open  loop  linear  regulator  control 
problem  (b-0)  starting  at  (s,x)  and  using  the  optimal  open  loop  control  for 
the  linear  regulator  problem  corresponding  to  initial  data  (0,xq)»  The 
process  x®(t)  represents  the  solution  to  the  above  Ito  stochastic 
differential  equation  with  b=0  and  use  of  the  optimal  linear  regulator 
control  as  the  control  function  in  the  system  dynamics. 
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Stationary  Stochastic  Control  and  Eigenvalue  Problems: 

The  principal  eigenvalue  for  linear  second  order  elliptic  equations 

with  natural  boundary  conditions  in  Stochastic  Analysis,  139-152. 

A.  Friedman  (Editor),  Academic  Press  (1978). 

A  minimum  principle  for  the  smallest  eigenvalue  for  second  order 

linear  elliptic  equations  with  natural  boundary  conditions,  Comm. 

Pure  Appl.  Math.  31(1978),  509-519. 

We  have  boon  able  to  derive  a  new  characterization  of  the  principal 
eigenvalue  for  second  order  linear  elliptic  partial  differential  equations, 
not  necessarily  self-adjoint,  wit1’  both  natural  and  Dirichlet  boundary 
conditions,  and  also  give  a  new  alternative  numerical  method  for 
calculating  both  the  principal  eigenvalue  and  corresponding  eigenvector  in 
the  case  of  natural  boundary  conditions.  The  principal  eigenvalue,  if 
appropriate  sign  changes  are  made,  determines  the  stability  of  equilibrium 
solutions  to  certain  second  order  nonlinear  partial  differential  equations. 
The  corresponding  eigenvector  enables  one  to  determine  the  first 
approximation  of  the  solution  of  the  nonlinear  equation  to  variations  of 
the  initial  condition  from  the  equilibrium  solution.  These  nonlinear 
equations  are  important  in  the  applications.  For  these  reasons  it  is 
important  to  have  these  characterizations  of  the  principal  eigenvalue  and 
eigenvector. 

Our  method  converts  the  determination  of  the  eigenvalue  and 


eigenvector  to  determining  the  solution  of  a  stationary  stochastic  control 


problem.  This  latter  problem  is  solved  and  from  it  a  numerical  scheme 
arises  naturally.  This  method  appears  to  have  applications  in  solving 
other  problems. 

Secondly,  progress  has  been  made  on  determining  the  asymptotic 
behavior  of  the  principal  eigenvalue  for  some  singularly  perturbed 
eigenvalue  problems  as  a  small  nuisance  parameter  tends  to  zero.  The 
principal  eigenvalue  is  the  optimal  value  for  a  singularly  perturbed 
stationary  stochastic  control  problem.  We  are  thus  able  to  determine  the 
asymptotic  behavior  of  the  optimal  value  of  certain  stationary  stochastic 
control  problems. 

Long-Term  Effects  of  Noise  on  Dynamical  Systems 

Stochastically  perturbed  limit  cycles,  J.  Applied  Probability 

15,  311-320  (1978). 

An  important  question  in  the  stability  and  control  of  stochastic 
systems  is  the  determination  of  the  limiting  long-time  behavior  of  the 
system  using  a  fixed  control.  This  work  answered  that  question  in  he  case 
of  a  stochastic  system  perturbed  by  a  small  additive  noise  term  where  the 
control  is  such  that  the  corresponding  deterministic  system  possesses  a 
stable  limit  cycle. 

It  is  shown  that  in  he  limit  of  large  time  the  stochastic  system  is 

4 

near  the  limit  cycle.  This  is  a  stability  result.  Moreover,  one  can 
compute  approximately  at  which  portions  of  the  limit  cycle  one  is  most 
likely  to  be  found.  Further  various  stationary  average  can  be  computed. 
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These  results  will  be  of  use  in  designing  approximate  controls  for 
stationary  stochastic  control  systems.  For  a  detailed  discussion  of  these 
results,  see  the  completed  above  paper. 


Asymptotic  Properties  of  Nonlinear  -  Diffusion  Equations. 


A  nonlinear  diffusion  problem  arising  in  plasma  physics,  (with  J. 
Berryman),  Phys.  Rev.  Letters,  4_0  (1978),  1720-1722. 

A  nonlinear  generali2ation  of  the  heat  equation  arising  in  plasma 
physics,  (with  J.  Berryman),  in  Applied  Nonlinear  Analysis,  61-66, 
V.  Lakshmikanthan  (Editor),  Academic  Press  (1979). 

Evolution  of  a  stable  profile  for  a  class  of  nonlinear  diffusion 
equations  with  fixed  boundaries,  (with  J.  Berryman),  J.  Math.  Phys. 
.19  (1978),  2476-2480. 

Stability  of  the  separable  solution  for  fast  diffusion,  (with  J. 
Berryman),  Arch.  Rat.  Mech.  Anal.,  74(1980),  379-388. 

A  technique  is  developed  for  studying  the  asymptotic  behavior  of 

certain  classes  of  systems  arising  in  the  applications  which  are  governed 

by  nonlinear  parabolic  partial  differential  equations.  The  techniques 

involved  the  construction  of  an  appropriate  Liapunov  type  function  in  the 

spirit  of  the  Liapunov  approach  for  ordinary  differential  equations.  It  is 

shown  that  under  certain  conditions  the  solution  of  the  nonlinear  parabolic 

equation  evolves  toward  an  appropriate  separable  solution  of  the  parabolic 

equation.  The  initial  condition  only  determines  the  asymptotic  amplitude 

of  the  separable  solution. 


The  first  paper  above  is  a  summary  of  the  application  of  this 
general  method  to  a  problem  in  plasma  physics  which  arises  in  studying  the 
diffusion  of  ionized  gas. 
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A  potential  weakness  of  the  approach  utilized  above  has  been  the 
inability  to  obtain  decay  rates  of  the  solution  to  the  equilibrium 
position  and  the  inability  to  verify  the  higher  order  approximations 
which  we  have  been  able  to  construct  formally.  We  have  worked,  without 
success,  in  this  area. 

Additionally,  we  have  studied  without  success  the  uniqueness 
question  of  the  limiting  equilibrium  position.  The  limiting  position 
is  described  by  the  equation  Au  +  ua  =  0,  u  =  0  on  3ft,  a  >  1. 


* 
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